{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "97c79c38-38a3-40f3-ba2e-250649347d63",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/run-llama/llama_cloud_services/blob/main/examples/parse/demo_starter_multimodal.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e081457",
   "metadata": {},
   "source": [
    "# Multimodal Parsing using LlamaParse\n",
    "\n",
    "This cookbook shows you how to use LlamaParse to parse any document with the multimodal capabilities of Multi-Modal LLMs from Anthropic/ OpenAI.\n",
    "\n",
    "LlamaParse allows you to plug in external, multimodal model vendors for parsing - we handle the error correction, validation, and scalability/reliability for you.\n",
    "\n",
    "Status:\n",
    "| Last Executed | Version | State      |\n",
    "|---------------|---------|------------|\n",
    "| Aug-19-2025   | 0.6.61  | Maintained |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "qOdqBxCS51Ow",
   "metadata": {},
   "source": [
    "### Installation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "H_Vqcylb50vm",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install llama-cloud-services"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15e60ecf-519c-41fc-911b-765adaf8bad4",
   "metadata": {},
   "source": [
    "### Setup\n",
    "\n",
    "Here we setup `LLAMA_CLOUD_API_KEY` for using `LlamaParse`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "91a9e532-1454-40e0-bbf0-fd442c350121",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# API access to llama-cloud\n",
    "os.environ[\"LLAMA_CLOUD_API_KEY\"] = \"llx-...\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "LGwBNPNotZRQ",
   "metadata": {},
   "source": [
    "## Download Data\n",
    "\n",
    "For this demonstration, we will use OpenAI's recent paper `Evaluation of OpenAI o1: Opportunities and Challenges of AGI`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "IjtKDQRLrylI",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget \"https://arxiv.org/pdf/2409.18486\" -O \"o1.pdf\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e29a9d7-5bd9-4fb8-8ec1-4c128a748662",
   "metadata": {},
   "source": [
    "## Initialize LlamaParse\n",
    "\n",
    "Initialize LlamaParse in multimodal mode, and specify the vendor.\n",
    "\n",
    "**NOTE**: optionally you can specify the Anthropic/ OpenAI API key. If you choose to do so LlamaParse will only charge you 1 credit (0.3c) per page. \n",
    "\n",
    "\n",
    "Using your own API key may incur additional costs from your model provider and could result in failed pages or documents if you do not have sufficient usage limits."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b5d6da6",
   "metadata": {},
   "source": [
    "### With anthropic-sonnet-4.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2e9d9cf-8189-4fcb-b34f-cde6cc0b59c8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Started parsing the file under job_id fdbe857e-48d0-4024-ba06-bfead78c4a0c\n"
     ]
    }
   ],
   "source": [
    "from llama_cloud_services import LlamaParse\n",
    "\n",
    "parser = LlamaParse(\n",
    "    # Enable pure multimodal parsing\n",
    "    parse_mode=\"parse_page_with_lvm\",\n",
    "    vendor_multimodal_model_name=\"anthropic-sonnet-4.0\",\n",
    "    # Pass in your own API key optionally\n",
    "    # vendor_multimodal_api_key=\"fake\",\n",
    "    target_pages=\"24\",\n",
    "    high_res_ocr=True,\n",
    "    adaptive_long_table=True,\n",
    "    outlined_table_extraction=True,\n",
    "    output_tables_as_HTML=True,\n",
    ")\n",
    "result = await parser.aparse(\"o1.pdf\")\n",
    "sonnet_nodes = result.get_markdown_nodes(split_by_page=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f3c51b0-7878-48d7-9bc3-02b516500128",
   "metadata": {},
   "source": [
    "### With GPT-4.1-mini\n",
    "\n",
    "For comparison, we will also parse the document using GPT-4.1-mini."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6fc3f258-50ae-4988-b904-c105463a498f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Started parsing the file under job_id faab19bf-0810-4437-a1ff-4f6ae36d6ce0\n"
     ]
    }
   ],
   "source": [
    "from llama_cloud_services import LlamaParse\n",
    "\n",
    "parser_gpt4o = LlamaParse(\n",
    "    # Enable pure multimodal parsing\n",
    "    parse_mode=\"parse_page_with_lvm\",\n",
    "    vendor_multimodal_model_name=\"openai-gpt-4-1-mini\",\n",
    "    # Pass in your own API key optionally\n",
    "    # vendor_multimodal_api_key=\"fake\",\n",
    "    target_pages=\"24\",\n",
    "    high_res_ocr=True,\n",
    "    adaptive_long_table=True,\n",
    "    outlined_table_extraction=True,\n",
    "    output_tables_as_HTML=True,\n",
    ")\n",
    "result = await parser_gpt4o.aparse(\"o1.pdf\")\n",
    "gpt_nodes = result.get_markdown_nodes(split_by_page=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44c20f7a-2901-4dd0-b635-a4b33c5664c1",
   "metadata": {},
   "source": [
    "### View Results\n",
    "\n",
    "Let's visualize the results along with the original document page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "778698aa-da7e-4081-b3b5-0372f228536f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "file_name: o1.pdf\n",
      "\n",
      "\n",
      "\n",
      "<table>\n",
      "<thead>\n",
      "<tr>\n",
      "<th>Participant_ID</th>\n",
      "<th>clinical Description Reference</th>\n",
      "</tr>\n",
      "</thead>\n",
      "<tbody>\n",
      "<tr>\n",
      "<td>Attribute</td>\n",
      "<td>Value</td>\n",
      "<td rowspan=\"12\"><strong>Basic Personal Information:</strong> Subject 098_S_0896 is a 72.0-year-old Female who has completed 15 years of education. The ethnicity is Not Hisp/Latino and race is White. Marital status is Married. Initially diagnosed as AD, as of the date 2007-10-24, the final diagnosis was Dementia.<br><br><strong>Biomarker Measurements:</strong> The subject's genetic profile includes an ApoE4 status of 0.0...<br><br><strong>Cognitive and Neurofunctional Assessments:</strong> The Mini-Mental State Examination score stands at 29.0. The Clinical Dementia Rating, sum of boxes, is 1.0. ADAS 11 and 13 scores are 4.67 and 4.67 respectively, with a score of 1.0 in delayed word recall...<br><br><strong>Volumetric Data:</strong> Under MRI conditions at a field strength of 1.5 Tesla MRI Tesla, using Cross-Sectional FreeSurfer (FreeSurfer Version 4.3), the imaging data recorded includes ventricles volume at 54422.0, hippocampus volume at 6717.0, whole brain volume at 1147980.0, entorhinal cortex volume at 2782.0, fusiform gyrus volume at 19432.0, and middle temporal area volume at 24951.0. The intracranial volume measured is 1799580.0....</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td>Age</td>\n",
      "<td>72.0</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td>Sex</td>\n",
      "<td>Female</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td>Education</td>\n",
      "<td>15</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td>Race</td>\n",
      "<td>White</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td>DX_bl</td>\n",
      "<td>AD</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td>DX</td>\n",
      "<td>Dementia</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td>...</td>\n",
      "<td>...</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td>APOE4</td>\n",
      "<td>1.0</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td>TAU</td>\n",
      "<td>212.5</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td>...</td>\n",
      "<td>...</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td>MMSE</td>\n",
      "<td>29.0</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td>CDRSB</td>\n",
      "<td>0.0</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td>...</td>\n",
      "<td>...</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td>FLDSTRENG</td>\n",
      "<td>1.5 Tesla MRI</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td>Ventricles</td>\n",
      "<td>84509</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td>Hippocampus</td>\n",
      "<td>5319</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td>...</td>\n",
      "<td>...</td>\n",
      "</tr>\n",
      "</tbody>\n",
      "</table>\n",
      "\n",
      "Figure 2: An example of a patient table and its corresponding clinical description.\n",
      "\n",
      "skills. Mathematics, as a highly structured and logic-driven discipline, provides an ideal testing ground for evaluating this reasoning ability. To investigate o1-preview's performance, we designed a series of tests covering various difficulty levels. We begin with high school-level math competition problems in this section, followed by college-level mathematics problems in the next section, allowing us to observe the model's logical reasoning across varying levels of complexity.\n",
      "\n",
      "In this section, we selected two primary areas of mathematics: algebra and counting and probability in this section. We chose these two topics because of their heavy reliance on problem-solving skills and their frequent use in assessing logical and abstract thinking [46]. The dataset used in testing is from the MATH dataset [46]. The problems in the dataset cover a wide range of subjects, including Prealgebra, Intermediate Algebra, Algebra, Geometry, Counting and Probability, Number Theory, and Precalculus. Each problem is categorized based on difficulty, ranked from level 1 to 5, according to the Art of Problem Solving (AoPS). The dataset mainly comprises problems from various high school math competitions, including the American Mathematics Competitions (AMC) 10 and 12, as well as the American Invitational Mathematics Examination (AIME), and other similar contests. Each problem comes with detailed reference solutions, allowing for a comprehensive comparison of o1-preview's solutions.\n",
      "\n",
      "In addition to evaluating the final answers produced by o1-preview, our analysis delves into the step-by-step reasoning process of the o1-preview's solutions. By comparing o1-preview's solutions with the dataset's solutions, we assess its ability to engage in logical reasoning, handle abstract problem-solving tasks, and apply structured approaches to reach correct answers. This deeper analysis offers insights into o1-preview's overall reasoning capabilities, using mathematics as a reliable indicator for logical and structured thought processes.\n",
      "\n",
      "25\n"
     ]
    }
   ],
   "source": [
    "# using Sonnet-4.0\n",
    "print(sonnet_nodes[0].get_content(metadata_mode=\"all\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1511a30f-3efc-4142-9668-7dc056a24d0c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "file_name: o1.pdf\n",
      "\n",
      "\n",
      "\n",
      "<table>\n",
      "<thead>\n",
      "<tr>\n",
      "<th colspan=\"2\"><b>Participant_ID</b></th>\n",
      "<th rowspan=\"2\" style=\"background-color: #b0b0b0;\"><b>clinical Description Reference</b></th>\n",
      "</tr>\n",
      "</thead>\n",
      "<tbody>\n",
      "<tr>\n",
      "<td><b>Attribute</b></td>\n",
      "<td><b>Value</b></td>\n",
      "<td rowspan=\"17\" style=\"background-color: #d0d0d0; vertical-align: top;\">\n",
      "<b>Basic Personal Information:</b> Subject 098_S_0896 is a 72.0-year-old Female who has completed 15 years of education. The ethnicity is Not Hisp/Latino and race is White. Marital status is Married. Initially diagnosed as AD, as of the date 2007-10-24, the final diagnosis was Dementia.<br><br>\n",
      "<b>Biomarker Measurements:</b> The subject's genetic profile includes an ApoE4 status of 0.0…<br><br>\n",
      "<b>Cognitive and Neurofunctional Assessments:</b> The Mini-Mental State Examination score stands at 29.0. The Clinical Dementia Rating, sum of boxes, is 1.0. ADAS 11 and 13 scores are 4.67 and 4.67 respectively, with a score of 1.0 in delayed word recall…<br><br>\n",
      "<b>Volumetric Data:</b> Under MRI conditions at a field strength of 1.5 Tesla MRI Tesla, using Cross-Sectional FreeSurfer (FreeSurfer Version 4.3), the imaging data recorded includes ventricles volume at 54422.0, hippocampus volume at 6677.0, whole brain volume at 1147980.0, entorhinal cortex volume at 2782.0, fusiform gyrus volume at 19432.0, and middle temporal area volume at 24951.0. The intracranial volume measured is 1799580.0.… \n",
      "</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td rowspan=\"7\"><b>Basic Personal information</b></td>\n",
      "<td>Age</td>\n",
      "<td>72.0</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td>Sex</td>\n",
      "<td>Female</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td>Education</td>\n",
      "<td>15</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td>Race</td>\n",
      "<td>White</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td>DX_bl</td>\n",
      "<td>AD</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td>DX</td>\n",
      "<td>Dementia</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td>…</td>\n",
      "<td>…</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td rowspan=\"3\"><b>Biomarker measurements</b></td>\n",
      "<td>APOE4</td>\n",
      "<td>1.0</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td>TAU</td>\n",
      "<td>212.5</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td>…</td>\n",
      "<td>…</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td rowspan=\"3\"><b>Cognitive and neurofunctional Assessments</b></td>\n",
      "<td>MMSE</td>\n",
      "<td>29.0</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td>CDRSB</td>\n",
      "<td>0.0</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td>…</td>\n",
      "<td>…</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td rowspan=\"4\"><b>Volumetric data</b></td>\n",
      "<td>FLDSTRENG</td>\n",
      "<td>1.5 Tesla MRI</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td>Ventricles</td>\n",
      "<td>84599</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td>Hippocampus</td>\n",
      "<td>5319</td>\n",
      "</tr>\n",
      "<tr>\n",
      "<td>…</td>\n",
      "<td>…</td>\n",
      "</tr>\n",
      "</tbody>\n",
      "</table>\n",
      "\n",
      "Figure 2: An example of a patient table and its corresponding clinical description.\n",
      "\n",
      "skills. Mathematics, as a highly structured and logic-driven discipline, provides an ideal testing ground for evaluating this reasoning ability. To investigate o1-preview’s performance, we designed a series of tests covering various difficulty levels. We begin with high school-level math competition problems in this section, followed by college-level mathematics problems in the next section, allowing us to observe the model’s logical reasoning across varying levels of complexity.\n",
      "\n",
      "In this section, we selected two primary areas of mathematics: algebra and counting and probability in this section. We chose these two topics because of their heavy reliance on problem-solving skills and their frequent use in assessing logical and abstract thinking [46]. The dataset used in testing is from the MATH dataset [46]. The problems in the dataset cover a wide range of subjects, including Prealgebra, Intermediate Algebra, Algebra, Geometry, Counting and Probability, Number Theory, and Precalculus. Each problem is categorized based on difficulty, ranked from level 1 to 5, according to the Art of Problem Solving (AoPS). The dataset mainly comprises problems from various high school math competitions, including the American Mathematics Competitions (AMC) 10 and 12, as well as the American Invitational Mathematics Examination (AIME), and other similar contests. Each problem comes with detailed reference solutions, allowing for a comprehensive comparison of o1-preview’s solutions.\n",
      "\n",
      "In addition to evaluating the final answers produced by o1-preview, our analysis delves into the step-by-step reasoning process of the o1-preview’s solutions. By comparing o1-preview’s solutions with the dataset’s solutions, we assess its ability to engage in logical reasoning, handle abstract problem-solving tasks, and apply structured approaches to reach correct answers. This deeper analysis offers insights into o1-preview’s overall reasoning capabilities, using mathematics as a reliable indicator for logical and structured thought processes.\n"
     ]
    }
   ],
   "source": [
    "# using GPT-4o\n",
    "print(gpt_nodes[0].get_content(metadata_mode=\"all\"))"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
